Feeling Stories: Enriching Story Listening Experience for Children with Haptic Feedback

Story listening contributes greatly to children’s development, and much research has explored different modalities of story listening in children. However, few studies have utilized haptic sensory input. In the present 2-session study, we implemented a haptic vest that generates haptic sensations on a child’s back as he or she listens to stories. In the first session, child participants rated how realistic a sensation was in comparison to a language phrase. In the second session, the same child participants listened to two stories with the vest on. At the end of each story, the child participants retold the story and then answered 9 comprehension questions. At the end of the session, the child participants indicated which one of the two stories they liked better. Our study showed that children of 6 years old had the ability to associate haptic sensations with semantic meanings. They could also distinguish between sensations with congruent semantic meanings and those with incongruent semantic meanings. From the second session of the study, we observed an increase in children’s comprehension of the stories when they felt story-relevant sensations from the vest. However, this was only true for the 5and 6-year olds. We did not find a similar effect in child participants’ retelling task. These results confirmed that 6-year olds had developed an association between haptics and semantics. In addition, sensations that were storyrelated improved children’s performance on story comprehension. We showed that haptic sensory input could improve story processing for children. FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 3 Feeling Stories: Enriching Story Listening Experience for Children with Haptic Feedback Story listening plays an important role in young children’s literacy development because their comprehension and story telling skills are boosted when linguistic components, such as grammar, vocabulary, and phrases are presented in a meaningful context (Glazer & Burke, 1994; Jennings, 1991). A study by Elley (1989) showed vocabulary gains in 7-year-old children after they listened to a story read to them. Follow-up tests showed that the gain was relatively longlasting. The same vocabulary gain was also shown to exist in kindergarteners after they listened to the same story twice (Robbins & Ehri, 1994). Using words unfamiliar to the kindergarteners in the story, researchers tested if the kindergarteners would recognize meanings of the words solely based on the context of the story. Robbins and Ehri found that children recognized meanings of significantly more of the unfamiliar words if they appeared in the stories compared to not in the stories. In addition, children who were exposed to story narratives tended to be better at understanding how stories were organized (Schickedanz, 1992). Beyond literacy development, story listening also influences young children’s value development. Berg-Cross and Berg-Cross (1978) found that children’s attitudes varied significantly after merely hearing stories that expressed values. In Berg-Cross’ study, children’s awareness of their stereotypical sex roles was weakened after listening to William’s Doll. Similarly, children were more willing to try something new after listening to the story Bread and Jam for Frances. These examples showed that after listening to books that contained social values, children of age 4 to six years old were prone to changing their expressed attitudes and values. Extending Berg-Cross and Berg-Cross’s work, many other researchers also found that FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 4 exposure to sex-equitable story books resulted in more flexible sex role attitudes for preschool children (Schau & Scott, 1984; Kropp & Halverson, 1983; Anderson & Many, 1992). Given the importance of story listening in young children, there has been much research on engaging non-auditory inputs for story processing. Visual components and physical interactions have been well explored. However, in comparison, there is a lack of research on using haptics in story listening. Visual components such as animation have been shown to positively affect children’s memory and comprehension of stories. A study by Beagles-Roos and Gat (1983) explored the effect of visuals on children of age 6.5 – 8 and 9 – 10.5 years old. The children were exposed to an animated audio/visual story and a visual story. Beagles-Roos and Gat found that children tended to recall more details of the story if it was presented in both auditory and visual modalities. The study also showed that the additional visuals moderately helped students recall details of stories they listened to. Another study done by Gibbon and colleagues (1986) presented similar results. Children of both 4 and seven years old produced more inferences when they saw actions in a story while listening to the story, compared to children who only listened to the story. The 4-year-olds also performed better at producing explicit information from the story they heard and saw. These results are evidence that visual inputs have a significant positive impact on children’s story memory and comprehension. Another new component to story listening for young children is physical interaction with digital books. Children can physically interact with the stories read to them by clicking on hotspots, objects on computer screens or e-books that trigger related animation or sound effects. However, it has not been demonstrated that such physical interactions can positively influence children’s story memory and comprehension. Ricci and Beal (2002) examined the effect of the FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 5 physical interaction on children’s story memory. In the study, first-grade children (6 – 7 years old) listened to the story The Ugly Duckling under 3 conditions. They listened to the story with audio only, with passive visual and audio, or with interactive visual components that they could click on. Ricci and Beal found that the additional interactions did not positively or negatively affect children’s memory and their ability to make inferences in comparison to passive visuals. Compared to the amount of research on visual and interactive story-listening technology, little research has explored the effect of haptics on story listening for young children. The use of haptics, defined as “the sensibility of the individual to the world adjacent to his body by use of his body” (Gibson, 1966, p. 97), is largely unexplored in children’s story-listening literature. One constraint that limits the research on this subject is that there is no clear evidence that children can comprehend haptic signals that convey meaningful information. However, evidence abounds on adults’ comprehension of such haptic signals. One of the earliest attempts to demonstrate meaningful haptic signal, i.e., haptic vocabulary, was Vibratese (Geldard, 1957). By corresponding forty-five basic numerals and letters to unique haptic patterns varied in amplitude, duration and location, Vibratese delivered abstract representations of content on an adult’s chest. Geldard showed that adults could haptically read and understood thirty-five words per minute. However, as groundbreaking as Vibratese was, this communication required a long training period for adults and had a slow learning curve. Since Vibratese, there have been many attempts to generate haptic signals with embedded semantic meanings. Some early attempts were based on two principles: pictorial mapping, which conveys spatial information to the blind, and frequency-place mapping (Tan & Pentland, 2001). However, attempts under the two principles remained time-consuming to train and use. Later, perceptually differentiable haptic vocabularies were introduced. While the haptic vocabularies FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 6 functioned as building blocks of complex haptic language, they were designed to be easily interpreted and learned. Examples of such haptic vocabulary were haptic phonemes (Enriquez, MacLean & Chita, 2006) and haptic icons (Ternes & MacLean, 2008). Recently, a new kind of haptic feedback was developed. It used tactile illusions to generate artificial haptic patterns associated with meaningful events, e.g., rain. Israr and his colleagues (submitted) developed a haptic pad that used such easy-to-understand tactile illusions to express semantic meanings. For instance, to simulate rain, the haptic pad would generate individual point vibrations on an adult’s back in a random pattern. The point vibrations gave people an illusion of feeling rain on their back. With the haptic pad, Israr and colleagues confirmed that adults had the ability to associate haptic patterns with semantic meanings. Under the guidance of the existing literature on haptics, there has been limited research on haptic technology for story reading in adults. One related work, Sensory Fiction, by the Media Lab at Massachusetts Institute of Technology, integrated simple haptics with story reading experience for adults (Heibeck, Hope & Legault, 2013). Wearing a vest, adults could feel artificial heartbeats that changed as the mood of the story varied. The vest also moderated temperature and pressure in sync with the progress of the story. However, technology akin to Sensory Fiction does not yet exist for children, producing a gap in research on haptic story-listening technology for young children. One necessary step towards such technology is to demonstrate that children, like adults, can associate haptic patterns with semantic meanings. This is one of the purposes of the present study. In this study, we tested a haptic vest for young children that could be used while listening to stories. The vest generates gentle vibrations on children’s back, simulating meaningful events such as rain, tap, etc. Using the vest, we investigated if children, like adults, have the ability to FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 7 associate haptic information with semantic meanings. Based on the results, we examined the effect of such haptic signals, delivered by the haptic vest, on children’s response to stories, including understanding, recall, and preferences. We tested three hypotheses: First, children should be able to associate a haptic signal with its intended semantic meaning. Second, the haptic signal from the vest will affect their memory for events in stories. By testing children aged 4, 5 and 6 years, we further investigated the age dependence of these effects, hypothesizing that children would perform better as they became older in age. Methods Participants. Twenty-nine children participated in the study. Ten were four years old; ten five years old; and nine six years old. Sixteen children were recruited through Carnegie Mellon University Children’s School of Psychology while the other thirteen children were recruited from Disney Children Participant Pool. Our Institutional Review Board approved both recruitment methods. The study is consisted of 2 parts, but only twenty-seven of the participants completed both. Part 1 Stimuli. Stimuli consisted of fifteen pairs of language phrases (LPs) and feel effects (FEs) that were directly adopted from a previous study on adults (Israr et al, submitted). FEs are haptic patterns generated using an array of vibratactile actuators. They are defined by SOA (stimulus onset asynchrony), duration, intensity, and ramp-up of each actuator. LPs are descriptions of the semantic content corresponding to the FEs. FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 8 The fifteen LPs chosen for this study were such that four-year-old children would be able to understand them. There were four types of FEs paired with the fifteen LPs. Core FEs were FEs that the majority of the adults agreed upon as the more appropriate representations of their respective LPs (e.g., “light rain” paired with a series of low-intensity taps). Mismatches consisted of Core FEs paired with LPs of different Core FEs (e.g., “light rain” paired with a single, intense tap). Synonyms were FEs of synonyms of the Core LP (e.g., “sprinkle” substituted for “light rain”). Finally, Inferences were FEs with parameter settings inferred from Core FEs using underlying semantic meanings of the LPs (e.g., settings between sprinkle and downpour used with the phrase “rain”). Of the fifteen LP-FE pairings, there were 5 Core FEs, 3 Synonyms, 1 Inference, and 6 Mismatches (see Table 1). Material. The stimuli were presented on a custom made vest for children of age 4 to six. The vest had an array of twenty high quality vibrotactile transducers, called tactors, embeded in it: five rows of four tactors across the back. The vest was connected to a laptop, which communicated with an iPad mini application through Wi-fi. The application allowed the experimenter to play FEs on the vest and input children’s responses. During the process of the study, children wore BOSE noise-cancelling headphones that masked motor noises produced by the tactors. Procedure. The Part 1 of the study is a shorter version of a previous adult study (Israr, submitted), where we found that LPs and FEs could reliably be paired. To test for corresponding effects with children, we used a similar study design. The experimenter first helped the child participant to put on the vest. The experiment adjusted the straps of the vest to make sure that the child participant’s back was in firm contact with the tactors. To confirm that the child participant could feel the tactors on the edges, as well FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 9 as to introduce the haptic sensation, the experimenter told the child participant that she would draw a line down his or her back. In reality, the line-drawing sensation was created by the edge tactors. Through vibration, they created a sensation of a continuously moving point. After drawing the line, the experimenter asked the child participant whether the line went down or across his or her back. This was repeated four times for each edge of the vest. After the child participant reported that they could feel all edge tactors, the experimenter instructed about the task. Specifically, the child participant had to decide whether or not a FE corresponded to a LP. The experimenter, who held the iPad mini at hand, read to the child participant a LP. Then, the experimenter played a FE and asked the child participant whether or not the FE felt like the LP. If the child participant responded yes, he or she then indicated how much the FE felt like the LP by responding either “a little” or “a lot”. If a child responded that the FE did not feel like the LPs, he or she would continue with the next trial without further questions. The experimenter recorded the responses on the iPad mini. All child participants rated all fifteen LP-FE pairs, which consisted of Core, Synonym, Inference, and Mismatch FEs. The four types were intermixed and presented to the child participants in a random order. As incentives for the children to answer the questions as well as giving the child participants an indication of how many questions there were left, the experimenter gave child participants stickers or stamps on a piece of label whenever they answered questions. The children could take the labels with them after the study. Before the child participant left after the Part 1 of the study study, the experimenter told the child that after a break he or she would return to Part 2 of the study where they would listen to two stories with the vest on. The experimenter then asked the child if there were any feelings FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 10 that he or she would want to feel in the story-listening study. To prompt responses, the child participant was provided with 3 options. The 3 prompts were “a dog licking your back”, “a bug landing on your back”, and “your mom patting you on your back”. The purpose of the questions was to motivate the children to associate FEs with LPs. Part 2 Stimuli. For the Part 2 of the study, we used 2 four-minute long stories that were written by a professional children novel writer for children of age 4 to six years olds. The 2 stories had different story lines but shared the same story arc. One of the stories was about a child explorer searching for a silver-striped tiger that legends said was as big as a building. After a series of small events, she found the tiger. But it was, to everyone’s surprise, small as a squirrel and very friendly to the child explorer. The other story took place in outer space. The main character was a package deliverer traveling by space ship, and the story was about a package that she delivered. Small incidents happened along the way but she reached her destination safely. To her surprise, she was greeted by a birthday party. Then she found out that the package she delivered was a birthday cake for herself. Both stories shared similar story events and matched word counts. They both had 7 identical FEs embedded in them but in different orders (see Table 2). These FEs were dispersed evenly in the story among the paragraphs and located at the same story events in both stories. The FEs were taken from the previous adult study but adapted in length of onset to fit the speed of story. The researchers also subjectively predetermined the time of onset and offset of the FEs relative to the story transcript. FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 11 We created 4 testing conditions and each child was randomly assigned to one of the four conditions. The conditions differed by story order and the FEs that were on (see Table 3). Even though both stories had 7 embedded FEs, only 4 of the 7 FEs were turned on in each story. Across the 4 conditions, two of the on-FEs were always the first and last FE of the story. The remaining two were pre-chosen by the researchers such that across all children and conditions, all FEs would be played for equal amount of times. The 4-condition design allowed us to have a balanced study design to compare children’s performance with and without the FEs with a small amount of available child participants. We also designed 9 comprehension questions for each story. Seven of the 9 questions were related to the seven story events associated with the FEs; the other 2 questions were derived from parts of the story that did not have FEs associated with them. Each question had 3 answer options, one of which was the correct answer. Material. The child participants wore the same vest and noise-cancelling headphones as in the previous study. Procedure. The Part 2 study was conducted after a break following the first study. The break was 24 to 48 hours long for children recruited from the Children’s School and 10 minutes for children recruited from Disney. As in Part 1, the experimenter first helped the child participant put on the vest and adjusted it to fit. Then, the experimenter used the vest to draw lines on the child participant’s back to make sure that it was fully contacting the child’s back. It also re-introduced the sensation to the child participant. After putting on and checking the vest, the experimenter instructed the child participant that, in the study, he or she would listen to two stories and answer a few questions after each FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 12 story about what he or she heard. As in the Part 1 study, the child participant could get stickers or stamps whenever he or she answered questions. After the instructions, the child participant put on the headphones and listened to the first story. Immediately afterwards, the experimenter asked the child to retell the story he or she heard. The child participant’s retelling was audio recorded for future data analysis. The purpose of the retelling task was to examine which parts of the story the child participant could recall from memory without any prompts and if his or her performance of recalling was related to the FEs. Following the story-retelling task were the 9 comprehension questions regarding the story. If the child participant asked for prompts or remained silent for more than 5 seconds, the experimenter offered the child the 3 answer options. If the child participant produced an answer not included in the 3 options, his or her response was recorded as “other response”. After answering the comprehension questions, the child participant moved on to the next story and repeated the same tasks for the second story. Upon finishing answering the 9 comprehension questions for the second story, the experimenter asked the child participant which one of the two stories he or she liked better. Results Part 1 To code the data, we converted the 3-point scale to numeric values. The “no” responses were coded as 0; “a little” as 1; and “a lot” as 2. The factors we incorporated in our study were age and the 4 FE cases: Core, Mismatch, Synonym, and Inference. We first performed a 2-way repeated ANOVA with Core/Synonym/Inference as within subject factor, and age as the between-subject factor. We found no significant difference between FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 13 the 3 cases, F(2, 25) = 1.63, p = 0.22. Furthermore, we did not find an interaction between age and cases, F(4, 52) = 1.30, p = 0.28. Because there was no difference between Core, Synonym, and Inference cases, where the underlying semantic meanings of the FEs matched with the LPs, we combined the 3 cases into one, called Matches. We then ran another 2-way repeated ANOVA with Matches/Mismatches as the within-subject factor, and age as the between-subject factor. We found a significant difference between the Matches and Mismatches, F(1, 26) = 4.50, p = 0.04. Children rated Matches significantly higher than Mismatches, M(Matches) = 1.10, SD(Matches) = 0.11, M(Mismatches) = 0.95, SD(Mismatches) = 0.11 (Figure 1). Although the 2-way ANOVA did not show an interaction between Matches/Mismatches and Age, F(2, 26) = 1.40, p = 0.27, the effect was clearly the strongest among the 6 year olds. Within-subject t-tests accordingly showed that only that age group reliably rated matches as higher than mismatches, p = .015. The average difference on the 3-point scale was 0.31 for 6year olds while it was 0.11 and 0.02 for 4and 5-year olds (Figure 2). Part 2 There were multiple measures for the Part 2 study, arising from the story comprehension questions, spontaneous story retell, and story preference question. For the story comprehension questions, a child received 1 point if he or she got the question correct and 0 points if he or she got it wrong. We first performed a 3-way repeated ANOVA with story and onset of FEs as the within-subject factors and age as the between-subject factor. The ANOVA showed a significant difference between the two stories, F(1, 24) = 8.883, p = 0.007. Then, we ran a t-test of the 2 stories and found a significant difference in children’s response accuracy between the two stories, p < 0.01. Child participants performed significantly FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 14 better in the Tiger story, M(Tiger) = 0.72, SD(Tiger) = 0.21, than the Space Story, M(Space) = 0.59, SD(Space) = 0.17 (Figure 3). To see if there was an age effect, we performed a within-subject t-test. For the Tiger story, the most important finding was a significant difference between the presence and absence of the FEs for children of 5and 6years old, t(16) = 2.87, p = 0.01 (Figure 4). But no significant difference was found when the 4 year olds were included, t(25) = 1.25, p = 0.22. For the Space story, we did not see a significant difference in children of any of the 3 age groups, t(25) = 0.14, p = 0.89, or the 5and 6year olds, t(16) = 0.65, p = 0.52. To analyze transcript from the spontaneous story retell, we asked two independent coders to segment the story transcripts into Communication units (C-units) that are independent clauses with their modifiers (Loban, 1976). The coders marked each C-unit to the corresponding line in the story. We then compared the responses and the kappa for the two coders was 0.92 (with the percentage of agreement of 96.58%). Aggregating the responses, we first determined that there was no difference between the amount of utterances children retold for the two stories, t(23) = 1.83, p = 0.08. There were 157 utterances in total for the Space story and 103 utterances for the Tiger story. Furthermore, we found a positive correlation across children between the number of details recalled from the Tiger story and in the Space story, r(22) = 0.77, p < 0.01 (Figure 5). In terms of age effect on story retelling, the 6 year olds recalled the most amount of details from the two stories, M(4) = 17, M(5) = 95, M(6) = 148. Additionally, t-tests showed that there was a significant difference between the 4and 6year olds, t(15) = 3.47, p < 0.01. However, the t-tests did not show a significant difference between the 4and 5-year olds, t(15) = 2.00, p = 0.06, 5and 6year olds, t(15) = 1.75, p = 0.10. FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 15 For the story preference question, we found that child participants’ story preference did not differ by age, X(4, N = 24) = 4.15, p =0.39. Discussion We proposed three hypotheses in this work. First, children should be able to associate a haptic signal with its intended semantic meaning. This was confirmed in our Part 1 study. Sixyear-old children rated FEs paired with their intended LPs much higher than FEs paired with LPs of other FEs. We believe that children of age of 6 could associate a haptic signal with its semantic meaning and distinguish between FEs paired with congruent LPs and with incongruent LPs. However, we did not observe such association in children of 4and 5years old. This might be because of the abstract nature of the task. Completing the task demands high mental resources that were not well developed in children of 4 and 5 years old. A possible future direction is to design a less demanding task for the 4and 5-year olds to confirm the presence of the association between semantics and haptics. Our second hypothesis was that the haptic signal from a haptic vest they wear would affect their memory for events in stories. And lastly, children would perform better in the hapticsemantic association task and the story memory task as they became older in age. From the Part 2 of the study, we found a significant difference in children’s performance in the comprehension questions between the Tiger story and the Space story. Since the children performed significantly better in the Tiger story than the Space story, we decided to look for effect of FEs on story comprehension in only the Tiger story. Disregarding children’s responses to the comprehension questions for the Space story, we found a significant improvement in performance for the 5and 6-year olds when the FEs were on FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 16 than when they were off. This confirms a positive effect of the FEs on the 5and 6-year olds memory for events in the story. Although the 4-year olds did not show such improvement in performance, they did not perform worse with the FEs on than with the FEs off. We believe that the equal performance showed that the FEs were not intrusive for the 4-year olds. On close examination of the comprehension questions for the Space story, we believe that the difference in performance between the two stories was a result of multiple factors. First of all, the comprehension questions for the Space story were not sufficiently designed to provide memory cues for the children to recall details of the stories, which were necessary to answer the questions. Also, the questions for the Space story were not as well connected to the storyline as the questions for the Tiger story were. In addition, the FEs used in the Space story were not as highly rated by adult participants from a previous study as the FEs used in the Tiger story. If the FEs could not trigger the underlying semantic meanings in children, they could not help them better comprehend the story. Future studies can modify the comprehension questions to facilitate recall. Also, the FEs should be carefully chosen so that the quality of the FEs remains the same across both stories. Somewhat surprisingly, the difference in performance for the comprehension questions between the Tiger and the Space story did not show up in the story-retelling task. Children retold similar amount of details in the two stories. In addition, children’s performance in retelling one story was positively correlated with their performance in retelling the other story. This supports our conclusion that children’s poor performance in the Space story comprehension questions was a result of the design of the questions and the relatively low quality of the FEs, instead of any difference between the two stories. FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 17 With respect to the number of details recalled during story retelling, we observed an age effect, such that the 6-year olds actively recalled more details from the stories than the 4-year olds. This is a common phenomenon where as the children grow older, their cognitive ability matures, which allows them to remember more details. Lastly, our data showed no effect of age on story preference. The child participants did not strongly prefer one story over the other, and this is true for children of all three ages. In this study, we found that children of 6 years old had the ability to associate a haptic signal with its underlying semantic meaning. Furthermore, for 5and 6-year olds, the presence of FEs could significantly improve their comprehension for stories they listened to. For children who did not show such improvement, their performance was not hampered by the onset of the FEs either, showing the unintrusive nature of the FEs. These results have implications for developing haptic story-listening technology that can help children better comprehend the stories read to them. Future research can also explore effect of sound on children’s perception of the FEs and how cooperating both sound effects and FEs can influence children’s story processing abilities. FEELING STORIES: ENRICHING STORY LISTENING EXPERIENCE FOR CHILDREN WITH HAPTIC FEEDBACK 18